<html>
<head><meta charset="utf-8"><title>crates.io rate limiting? · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F.html">crates.io rate limiting?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="236346849"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20rate%20limiting%3F/near/236346849" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Simon Sapin <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F.html#236346849">(Apr 27 2021 at 14:30)</a>:</h4>
<p>Hello. <a href="https://crates.io/policies#crawlers">https://crates.io/policies#crawlers</a> mentions a 1 request per second rate limit, does that also apply to downloading crate tarballs? (Which it technically an API endpoint among others)</p>



<a name="236346902"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20rate%20limiting%3F/near/236346902" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Simon Sapin <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F.html#236346902">(Apr 27 2021 at 14:30)</a>:</h4>
<p>For the purpose of downloading many crates</p>



<a name="236346921"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20rate%20limiting%3F/near/236346921" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Simon Sapin <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F.html#236346921">(Apr 27 2021 at 14:30)</a>:</h4>
<p>up to all of them</p>



<a name="236347315"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20rate%20limiting%3F/near/236347315" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Simon Sapin <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F.html#236347315">(Apr 27 2021 at 14:32)</a>:</h4>
<p>I see <a href="https://www.pietroalbini.org/blog/downloading-crates-io/">https://www.pietroalbini.org/blog/downloading-crates-io/</a> recommends skipping the API and requesting directly from <a href="http://static.crates.io">static.crates.io</a></p>



<a name="236348227"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20rate%20limiting%3F/near/236348227" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Simon Sapin <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F.html#236348227">(Apr 27 2021 at 14:37)</a>:</h4>
<p>How stable is that URL?</p>



<a name="236349975"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20rate%20limiting%3F/near/236349975" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F.html#236349975">(Apr 27 2021 at 14:47)</a>:</h4>
<p>The <a href="http://static.crates.io">static.crates.io</a> URL format hasn't changed and I wouldn't expect it to</p>



<a name="236359777"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20rate%20limiting%3F/near/236359777" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F.html#236359777">(Apr 27 2021 at 15:33)</a>:</h4>
<p><span class="user-mention silent" data-user-id="219747">Simon Sapin</span> <a href="#narrow/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F/near/236346849">said</a>:</p>
<blockquote>
<p>Hello. <a href="https://crates.io/policies#crawlers">https://crates.io/policies#crawlers</a> mentions a 1 request per second rate limit, does that also apply to downloading crate tarballs? (Which it technically an API endpoint among others)</p>
</blockquote>
<p>we don't really enforce the rate limit on the download endpoint</p>



<a name="236359838"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20rate%20limiting%3F/near/236359838" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F.html#236359838">(Apr 27 2021 at 15:33)</a>:</h4>
<p>still, if you need to download all the crates it'd be better to hit <a href="http://static.creates.io">static.creates.io</a> directly</p>



<a name="236359943"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20rate%20limiting%3F/near/236359943" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F.html#236359943">(Apr 27 2021 at 15:34)</a>:</h4>
<p>but you can hit the download endpoint</p>



<a name="236359987"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20rate%20limiting%3F/near/236359987" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F.html#236359987">(Apr 27 2021 at 15:34)</a>:</h4>
<p>just... please don't single-handedly send 1k requests/second :)</p>



<a name="236360557"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20rate%20limiting%3F/near/236360557" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F.html#236360557">(Apr 27 2021 at 15:37)</a>:</h4>
<p>If you hit the download endpoint it will count as a download in the crate statistics while going directly through <a href="http://static.crates.io">static.crates.io</a> it doesn't count as far as I know.</p>



<a name="236412739"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/crates.io%20rate%20limiting%3F/near/236412739" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Simon Sapin <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F.html#236412739">(Apr 27 2021 at 21:22)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/crates.2Eio.20rate.20limiting.3F/near/236359777">said</a>:</p>
<blockquote>
<p>we don't really enforce the rate limit on the download endpoint</p>
</blockquote>
<p>I kinda suspected that, but I’d still prefer we act as "good citizens" :)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>